---
id: pydanticai
title: Pydantic AI
sidebar_label: Pydantic AI
---

import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
import { Timeline, TimelineItem } from "@site/src/components/Timeline";
import VideoDisplayer from "@site/src/components/VideoDisplayer";
import ColabButton from "@site/src/components/ColabButton";

# Pydantic AI

[Pydantic AI](https://ai.pydantic.dev/) is a Python framework for building reliable, production-grade applications with Generative AI, providing type safety and validation for agent outputs and LLM interactions.

:::tip
We recommend logging in to [Confident AI](https://app.confident-ai.com) to view your Pydantic AI evaluations.

```bash
deepeval login
```

:::

## End-to-End Evals

`deepeval` allows you to evaluate Pydantic AI agents **under a minute**.

<Timeline>

<TimelineItem title="Configure Pydantic AI">

Pass `agent_metrics` to the `ConfidentInstrumentationSettings` constructor.

```python title="main.py" showLineNumbers
from pydantic_ai import Agent
from deepeval.integrations.pydantic_ai.instrumentator import (
    ConfidentInstrumentationSettings,
)
from deepeval.metrics import AnswerRelevancyMetric

agent = Agent(
    "openai:gpt-5",
    instructions="You are a helpful assistant.",
    instrument=ConfidentInstrumentationSettings(
        is_test_mode=True,
        agent_metrics=[AnswerRelevancyMetric()]
    ),
)
```

:::info
Evaluations are supported for Pydantic AI `Agent`. Only metrics with parameters `input`, `output` and `tools_called` are eligible for evaluation.
:::

</TimelineItem>
<TimelineItem title="Run evaluations">

Create an `EvaluationDataset` and invoke your Pydantic AI application for each golden within the `evals_iterator()` loop to run end-to-end evaluations.

<Tabs groupId="pydantic_ai">
<TabItem value="asynchronous" label="Asynchronous">

```python title="main.py" showLineNumbers
import asyncio

dataset = EvaluationDataset(
    goldens=[
        Golden(input="What's the weather in Paris?"),
        Golden(input="What's the weather in London?"),
    ]
)

for golden in dataset.evals_iterator():
    task = asyncio.create_task(run_agent(golden.input))
    dataset.evaluate(task)
```

</TabItem>
</Tabs>

✅ Done. The `evals_iterator` will automatically generate a test run with individual evaluation traces for each golden.

</TimelineItem>
<TimelineItem title="View on Confident AI (optional)">

<VideoDisplayer
  src="https://confident-bucket.s3.us-east-1.amazonaws.com/end-to-end%3Apydantic-1080.mp4"
/>

</TimelineItem>

</Timeline>

:::note
If you need to evaluate individual components of your Pydantic AI application, [set up tracing](/docs/evaluation-llm-tracing) instead.
:::


## Evals in Production

To run online evaluations in production, replace `metrics` with a [metric collection](https://documentation.confident-ai.com/docs/llm-tracing/evaluations#online-evaluations) string from Confident AI, and push your Pydantic AI agent to production.

```python filename="main.py" showLineNumbers
from pydantic_ai import Agent
from deepeval.integrations.pydantic_ai import ConfidentInstrumentationSettings

agent = Agent(
    "openai:gpt-4o-mini",
    system_prompt="Be concise, reply with one sentence.",
    instrument=ConfidentInstrumentationSettings(
        agent_metric_collection="test_collection_1",
    )
)

result = agent.run_sync(
    "What are the LLMs?"
)
```
